Inspiration

• Kentucky’s accountability system re-design decision to forgo growth and utilize improvement instead.
• Countless discussions at TAC meetings where accountability system terminology is used interchangeably and incorrectly (most notably by experts in our field).
• Numerous presentations at TAC meetings discussing poorly designed quasi-experimental studies using results from state summative assessments.
• Characterizations of “learning loss” and “recovery” during and after the COVID-19 pandemic.
• Frequent staff meeting discussions about accountability systems and their design.
• 2023 Brian Gong Colloquium on accountability system design.

The Reign of Proficiency

• During the last week of January, National Assessment of Educational Progress (NAEP) results were released.
• If you read beyond the headlines, an implicit message about communicating the performance of our education system is crystal clear: Proficiency reigns supreme.
• Since at least the NCLB era, education policy has been fixated on proficiency levels and their trends (i.e., achievement/attainment).
• Call it what you will —- proficiency, mastery, college and career readiness -— student attainment is the primary measure of educational performance.
• Proficiency is King! And it should be!
• Why? Claiming an education system is “good” when only a small fraction of students are proficient is absurd.
• But here’s the catch: Proficiency and proficiency trends are the wrong tools for the effective performance management necessary to improve an education system.
• Death to the King! Long Live the King!

Death to Proficiency!

• Since at least the No Child Left Behind (NCLB) era, attainment has had its detractors.
• Rightly, these critics have pointed out that using attainment for accountability purposes (e.g., school ratings) is problematic due to selection bias – students are not randomly assigned to schools.
• The solution to this problem has been to incorporate growth or value-added into the accountability system.
• Over the last 15 years, one of the primary foci of accountability system design has been weighting attainment and growth components to create more “valid” accountability indicator systems.
  
• The result, I would argue, is a design process that treats these two components like food items at a buffet.
• And today several efforts to create more valid accountability systems promote adding more components.
• The result of this process of adding more components, I contend, is incoherence in the design of accountability systems.

Attainment & Learning

• And as indicators used for accountability have increased, a significant part of the ensuing incoherence has been the imprecise, inconsistent, and incorrect use of terminology.
• Just considering results from large-scale assessments, the number of different terms is daunting:
  • Achievement, proficiency, attainment, status, and mastery are several terms frequently encountered to discuss what I refer to as attainment, a characterization of what a student or groups of students know and can do at a point in time.
  • Growth, value-added, learning, SGPs, value-tables, content-referenced growth are several terms used to characterize student growth of a student or group of students.
  • Improvement is a term that seems to be used to describe a beneficial change (either for a student or a group of students).
  • Acceleration is a term that came to prominence during the pandemic to motivate goals to correct learning loss by accelerating student learning.
• Having attended hundreds of meetings and presentations over the last decade, I can’t count the number of times I’ve heard several of these terms used in ways that have led me to believe there is a fundamental lack of understanding about how large scale assessment results should be used.

Attainment & Learning

• Another example of of problematic terminology is the phrase “Assessment of learning”.
• Its problematic nature, however, helps illustrate a relationship necessary to begin fixing the problem.
• Learning is an unobserved process that occurs over time in the mind of the student.
  
• Assessments measure (or indicate) what a student knows and can do at a point in time.
• We infer learning by observing attainment (either formally or informally) over time.
  • Learning/Growth   is a leading indicator of attainment (SGPs, for example, are a coarse quantification of learning).
  • Attainment   is a lagging indicator of learning/growth.

Coherence in Accountability System Design

• There is no shortage of criticism of current accountability systems.
• A substantial part of many criticisms is the over-reliance on the results from large-scale assessments.
• I would argue that the root of the problem isn’t the over-reliance on the results from large-scale assessments, but rather the incoherent way in which results from large-scale assessments are utilized.
• Accountability systems are the result of policy initiatives: A coherent accountability system ideally connects the larger policy goals of the education system (increased rates of proficiency and the closing of achievement gaps) with the processes associated with achieving (or not achieving) those goals.
• Even with coherence, there is no guarantee that the education system will achieve the policy goals.
• But without coherence, I contend, accountability initiatives intended to improve the education system are doomed to fail.

The Solution

• In the following slides I will discuss a framework for accountability systems that I believe is more coherent and effective.
• Beginning with a formal discussion of terminology so that there is a shared understanding of the terms used, I will discuss how these components fit together.
• Once we understand how the components fit together, I will discuss the analytic framework sgpFlow that connects assessment results from large-scale assessments in a coherent fashion.
• Despite just beginning to investigate this framework, I believe it has the potential to transform performance management in education and the accountability systems that support that effort.

Coherence

• The term coherence is not frequently used in discussions of accountability systems and when it is, it is often not clearly defined.
• What is coherence with respect to accountability systems?
• What is incoherent about current accountability systems?
• Why is coherence important?
• How can coherence be instantiated in accountability systems?
• What benefits does coherence provide?

Gill’s Framework

• Brian Gill’s (2022) paper, What should the future of educational accountability look like?, introduces a useful framework for indicators used in accountability systems.
• Three Core Components of Gill’s Framework
  • Processes: The activities, instructional practices, and organizational elements that schools implement to support student learning.
  • Impacts: The measurable effects of processes on student learning and development, often captured through growth metrics or value-added models.
  • Outcomes: The long-term results of education, such as achievement/proficiency levels, graduation rates, and other indicators of student success.
• Gill argues that for indicators to drive meaningful improvement, they must be:
  • Valid: Accurately measure what they claim to measure without systematic bias.
  • Reliable: Stable and consistent over time to avoid misdiagnosis or loss of credibility.
  • Robust: Resistant to unintended consequences and manipulation while capturing critical dimensions of educational quality.

Gill’s Infographic

Coherent Components

Why Coherence is Critical?

What is Equity?